Can Clustered File Systems Support Data Intensive Applications?

This WIP attempts to address the question—Can cluster file systems match specialized file systems such as Google's GFS for data-intensive applications? With the explosive growth of information and applications exploiting that information, large-scale data processing has emerged as an important challenge. Example applications include web search, indexing and mining, discovering biological functions from genomic sequences, astronomical phenomena from telescope imagery, and brain-scale networks for cog-nitive systems. These data-intensive applications demand a scalable, yet cost-effective storage layer. The storage layer for these applications needs to scale to thousands of nodes so that highly parallel applications process petabytes of data in hours rather than days. At the same time, the infrastructure needs to be built on commodity components to minimize cost while tolerating the failures that are typical for these components. Given the large volumes of data being processed, another key requirement for this storage layer is to enable shipping compute to data rather than the other way around. Recently, enterprises faced with these critical needs of data-intensive applications have proposed specialized file systems, built with the unique requirements of the layer in mind. For example , Google developed the GFS file system that is optimized for large sequential and small random reads on a small number of large files residing on a commodity cluster. Companies such as Yahoo and Kosmix followed this trend by emulating the GFS architecture in Hadoop DFS and KFS respectively. For the scope of this work, we choose the open source Hadoop DFS (HDFS) as a representative specialized file system. This work argues that cluster file systems can also rise to the challenges posed by these data-intensive applications. Moreover, there are inherent advantages to using cluster file systems in this paradigm: (1) These file systems can provide well-known traditional file APIs to these new class of applications. (2) Given that these file systems have been around for a while, they are enabled with a rich set of management tools such as automated backup and disaster recovery etc. (3) These file systems can simultaneously support legacy applications that rely on traditional file APIs obviating the need to maintain different storage layers for different applications. (4) Finally, an interesting trend further motivates this study: enterprises are increasingly incorporating data analytics in their workflows resulting in a mix of legacy and the new class of data-intensive applications accessing a common storage layer. There is ample evidence that existing cluster file …